Skip to main content

Fluentd

Fluentd Logo

Fluentd is an open source data collector for building the unified logging layer. Once installed on a server, it runs in the background to collect, parse, transform, analyze and store various types of data. For most small to medium sized deployments, fluentd is fast and consumes relatively minimal resources.

How does it work?

Fluentd scraps logs from a given set of sources, processes them (converting into a structured data format) and then forwards them to other services like Elasticsearch, object storage etc. Fluentd is especially flexible when it comes to integrations – it works with 300+ log storage and analytic services.

  • Fluentd gets data from multiple sources.

  • It structures and tags data.

  • It then sends the data to multiple destinations, based on matching tags

Fluentd Logo

Architecture:

  • Unified Logging with JSON: Fluentd tries to structure data as JSON as much as possible: this allows Fluentd to unify all facets of processing log data: collecting, filtering, buffering, and outputting logs across multiple sources and destinations (Unified Logging Layer).

  • Pluggable Architecture: Fluentd has a flexible plugin system that allows the community to extend its functionality. Our 500+ community-contributed plugins connect dozens of data sources and data outputs. By leveraging the plugins, you can start making better use of your logs right away.

  • Minimum Resources Required: Fluentd is written in a combination of C language and Ruby, and requires very little system resource. The vanilla instance runs on 30-40MB of memory and can process 13,000 events/second/core.

  • Built-in Reliability: Fluentd supports memory- and file-based buffering to prevent inter-node data loss. Fluentd also supports robust failover and can be set up for high availability

Benefits of Fluentd :

  • Unified Logging Layer: Fluentd decouples data sources from backend systems by providing a unified logging layer in between.

  • Simple and easy yet flexible: Fluentd's 500+ plugins makes it compatible with dozens of data sources and data outputs. Plugins are easy to write and deploy, too.

  • Open Source: Fluentd is Apache2.0 licensed, fully open source software. That means your imagination, not license restrictions, is the limit of what you can achieve with Fluentd. The source code is available on GitHub.

  • Proven Reliability and Performance: 5000+ data driven companies rely on Fluentd. Some Fluentd users collect data from thousands of machines in real-time. Thanks to its small memory footprint (30~40MB), you can save a lot of memory at scale.

  • Community: The community behind Fluentd keeps improving the software and helps one another to make Fluentd useful for everyone.